Doing Practical Data Science

Overview

• A Few Definitions: Data Science & Data Scientist
• A Few Real-World Examples: Data Analysis v.s. Data Science
• A Few Thoughts: On Building a DS Team
• A Few Questions to ask: On Recruiting

A little bit about myself

Academically…

• Did a lot of math & proofs back in school days
• Mostly dealt with papers and pencils instead of computers
• Started to get more involved in Stat & CS in grad school
• Stumbled upon the field of Data Science by accident

Professionally…

• Analytics Engineer @Washington Post Labs
• Data Scientist on Growth Team @Twitter

What is a Data Scientist?

There are many open definitions:

• "A data scientist is a data analyst who lives in San Franciso, California"
• "A data scientist is a business analyst who lives in New York"
• "A data scientist is someone who doesn’t use version control"

My Favorite Definition

Credit: Josh Wills

The Most Famous Venn Diagram in Data Science

Credit:Drew Conway

What Do Data Scientists Do Exactly?

Credit:Thomas Nguyen

Defining Key Performance Metrics

All startups and companies want to make money, become cash positive, and see millions of users using their services. We need to measure against ourselves

• User Engagement: Time in app, User Active Minutes, #Tweets produced, photos/videos uploaded
• Growth: MAU/DAU, Resurrection/Churn rates…etc

Data Warehousing

Credit:Twitter Data Platform Team

Data Analysis

Credit: Growth Data Science Team

Hard Core Data Science

Credit: Growth Data Science Team

Data Analysis v.s. Data Science, Go!

• Data Analysis
  • Shoter, Ad-hoc, and fast turnaround in nature
  • Multiple scripts per day, iterate until results look good, rarely code reviewed
  • Scalability and code readability is rarely a concern
  • Goal is to provide answers and insights quickly for decision making
• Data Science
  • Longer term, more math and engineering thinking involved
  • Often need to be productionized to take it to completion
  • Always code reviewed, Long-term maintainability is essential
  • Goal is to create tools to solve the next N problems, not just the current one

Data Analysis: An Example

• We don’t have much visibility into our Monthly Active Users Base
  • Are active users mostly resurrected users, or organic new Twitter users?
• We don’t know why Growth has exploded in one month, while it has stagnated in another
  • Are there seasonal or cyclical patterns to our user growth?
• We don’t know where Growth is coming from
  • Are they from Growth Marketing, better product features, or tactical works?
• We don’t know how sticky resurrected users are
  • What is the login day distribution of our resurrected users?

Answering the Question at hand

How many users churned in each country on 03/01/2015?

select
    country_id, count(distinct user_id)
from
(
SELECT
        '2015-03-01' AS created_date, user_id, country_id, deactivated
        FROM growth_staging.full_compact_snapshot_staging
        WHERE created_date = '2015-03-01'
              AND NOT today_mau
              AND yday_mau
              AND NOT suspended
) churned_mau_daily
where created_date = '2015-03-01' and deactivated = 0
group by country_id
order by country_id;

Ok, but this only answered one question, not very efficient

Answering the next N questions

Tables created over time

Some thoughts on Data Analysis

• Data asks and data pulls are surprisingly frequent — requests arrive all the time
• It would be extremely inefficient to process them one by one, and it also quickly becomes boring
• It’s important to think about how to answer the next N questions instead of just the question at hand
• The key is to build processes around it — data sources, pipeline, or dashboards
• Make the process as automated and self-serve as possible

Moving on to Data Science…

Data Science: An Example

• Ok, we need to reduce churn. How?
• Work with PMs to brainstorm product ideas and opportunity size
• Insights from Eng: Regardless of the use cases, we need to be able to target at-risk users efficiently
• We have targeting capabilities that are rule based, ad-hoc in nature
• Let’s build a model that can assign a churn score for each user
• User Health Service - store the k,v pairs (k=user_id, v=churn scores) into our key value store
• All internal services can consume it easily :)

Machine Learning is Founded in Math

Credit: Jimmy Lin & Dimitriy Ryaboy

Machine Learning != R/Python script

Credit: Jimmy Lin & Dimitriy Ryaboy

Machine Learning is Expressed in Codes

Credit: Jimmy Lin & Dimitriy Ryaboy

Bias to Productionization

Credit: Jimmy Lin & Dimitriy Ryaboy

ML fits well with the Map-Reduce Paradigm

Credit: Jimmy Lin & Dimitriy Ryaboy

How everything fits together

Credit: Jimmy Lin & Dimitriy Ryaboy

Some Thoughts on Data Science

• "Machine learning is not equivalent to R scripts. ML is founded in math, expressed in code, and assembled into software." - Ian Wong, former DS @Square
• “if you don’t write production code then you’re not productive.”
• A super fine tuned model on a laptop << A simple model in production
• Always start from a simple model, and iterate on it
• Doing Data Science is hard - Mikio L. Braun
  • Building a model is so easy to get wrong
  • It’s too easy to lie to yourself about it working
  • It’s very hard to tell whether it could work if it doesn’t
  • There is no free lunch

Experimentation & A/B Tests

• Defining the Hypotheses
• Desinging the control and experiment buckets
• Work out the design of the experiment to avoid bias
• Do we have enough sample size? What is the statistical power of the test?
• Do we have the proper scribes to track success metrics
• Write the experiment code
• Launch
• Analysis & Post-hoc Analysis

Experimentation & A/B Tests

An example of Experiment Brief

Experimentation & A/B Tests

An example of our A/B Tests Framework

Recap: What do data scientists do?

• We do a bunch of ad-hoc analyses
• We build data processes to make our lives (and others') easier
  • Data sources
  • Data pipelines
  • Data dashboards
• We build predictive model to optimize
• We do expeirment designs and run A/B tests
• We communicate
  • With execs and leadership on KPIs and OKRs
  • With PMs to help them to opportunity size and scoep their product features
  • With Engineering Team and discuss what and how to build things

Building a Data Science Team

• First understand the Analytics Need of the Organization
• Hire according to the needs
• Having a strong data platform team is the key to doing data science
• Once the foundation is solid, hire specialists who can optimize a particular area
• Bias toward hiring those who are not afraid of navigating the engineering world

Recruiting

• What is the charter of the data science team?
  • Give a clear, concise statement of purpose
• Is this an production engineering role? Analyst role?
  • We should be very clear about these distinctions up front
• How does the team interact with the rest of the company?
  • Work with Product Managers, Embedded on an Engineering Team, or in DS silos?
• Walk through a process for seeing an idea from the team implemented in production
  • Give well-defined series of steps
• What’s the data stack?
  • How do we manage and interact with our compute resources
• WHat kind of professional and educational opportunities are available?
  • STRATA+Hadoop, KDD, ICML, NIPS

Thank you!

• Please follow me on Twitter at @_rchang!